New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move trainer.step() before metric.update to overlap between backward pass and allreduce #1609

Merged

zhreshold merged 1 commit into dmlc:master from karan6181:overlap

Feb 10, 2021

Contributor

karan6181 commented Feb 10, 2021

The metric.update() is a sync call and it was preventing the overlap between communication and computation since it was present between loss.backward() and trainer.step().
Moved the metric.update() after trainer.step() to overlap communication and computation. Hence, the correct order is forward pass -> calculate loss -> backward pass -> allreduce -> metric update.
Also fixed minor error in the train_mask_rcnn.py script.


          Move trainer.step() before metric.update to overlap between backward …

a88e896

…pass and allreduce

github-actions bot commented Feb 10, 2021

Job PR-1609-a88e896 is done.
Docs are uploaded to http://gluon-vision-staging.s3-website-us-west-2.amazonaws.com/PR-1609/a88e896/index.html

zhreshold approved these changes

View reviewed changes

zhreshold merged commit 1ff5446 into dmlc:master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet